test(antithesis): enable forwarder disk persistence by blt · Pull Request #1800 · DataDog/saluki

blt · 2026-06-02T15:59:39Z

Summary

Sample forwarder_storage_max_size_in_bytes 50/50 on/off with forwarder_storage_path on a persistent compose volume, so the on-disk retry queue and restart-recovery paths run.

Bug discovered, with persistence on, a network partition fills the disk-backed retry queue, and the forwarder logs error! per failed retry attempt. The size of this spew is relative to the size of the inbound backlog.

Change Type

Bug fix
New feature
Non-functional (chore, refactoring, docs)
Performance

How did you test this PR?

References

N/A

blt · 2026-06-02T15:59:57Z

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

datadog-prod-us1-4 · 2026-06-02T16:00:51Z

✨ Fix all issues with BitsAI

⚠️ Warnings

🚦 1 Pipeline job failed

Semantic PR Title Check | Check For Semantic PR Title

🛟 This job is unlikely to succeed on retry. Please review your pipeline configuration.
Unknown release type "test" found in pull request title "test(antithesis): enable forwarder disk persistence".

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 564e457 | Docs | Datadog PR Page | Give us feedback!}

pr-commenter · 2026-06-02T16:07:41Z

Binary Size Analysis (Agent Data Plane)

Baseline: dc62bbd · Comparison: 564e457 · diff
Analysis Configuration: stripped binaries · Pass/Fail Threshold: +5%
Sizes: 37.85 MiB (baseline) vs 37.92 MiB (comparison)
Size Change: +70.58 KiB (+0.18%)

✅ Binary size difference within threshold

Changes by Module

Module	File Size	Symbols
`tonic`	-27.67 KiB	207
`tokio`	+25.30 KiB	878
`alloc`	+17.64 KiB	449
`[sections]`	+16.64 KiB	6
`figment`	-16.22 KiB	159
`comfy_table`	+16.17 KiB	9
`agent_data_plane::main::_{{closure}}`	-15.61 KiB	1
`chrono`	-12.27 KiB	10
`http_body_util`	+11.95 KiB	47
`serde_core`	+11.13 KiB	80
`agent_data_plane::internal::env`	+10.50 KiB	26
`hyper_timeout`	+8.77 KiB	8
`tower_layer`	+6.56 KiB	13
`serde_json`	-5.87 KiB	50
`agent_data_plane::components::ottl_transform_processor`	+4.53 KiB	10
`anyhow`	+4.47 KiB	455
`anon.fc864d48bd6fb31b92d502387c127e70.164.llvm.6438344938255192491`	-3.90 KiB	1
`agent_data_plane::cli::run`	-3.86 KiB	6
`anon.ac13b3cffb0e984c8eadac899797a3c1.67.llvm.8878368649288951961`	+3.81 KiB	1
`futures_channel`	+2.74 KiB	48

Detailed Symbol Changes

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  +0.5% +55.3Ki  +0.7% +58.0Ki    [12031 Others]
   +51% +20.2Ki   +51% +20.2Ki    agent_data_plane::internal::env::workload::build_collector::_{{closure}}::h48c3bde421c3ca9e
   +52% +19.8Ki   +52% +19.8Ki    agent_data_plane::internal::env::ADPEnvironmentProvider::from_configuration::_{{closure}}::h9670fb6d171f7edc
  [NEW] +17.8Ki  [NEW] +17.7Ki    tokio::runtime::park::CachedParkThread::block_on::hf364174df10d3359
 +34e2% +16.2Ki +41e2% +16.2Ki    comfy_table::cell::Cell::new::hbfd14b14cc593aba
  +1.4% +15.3Ki  +1.4% +15.3Ki    [section .gcc_except_table]
  [NEW] +4.87Ki  [NEW] +4.61Ki    agent_data_plane::components::ottl_transform_processor::config::_::_<impl serde_core::de::Deserialize for agent_data_plane::components::ottl_transform_processor::config::OttlTransformConfig>::deserialize::h6879914fbe295bfb
  [NEW] +4.23Ki  [NEW] +4.10Ki    agent_data_plane::state::metrics::rules::aggregation::get_aggregation_remappings::h8df5781a6f761921
   +67% +3.93Ki   +68% +3.93Ki    h2::proto::connection::Connection<T,P,B>::poll::hf5517bad6e618457
  +689% +3.88Ki  [ = ]       0    core::ptr::drop_in_place<http_body_util::combinators::map_err::MapErr<tonic::body::Body,axum_core::error::Error::new<tonic::status::Status>>>::heab8229a141536d3
  [NEW] +3.87Ki  [NEW] +3.72Ki    _<core::marker::PhantomData<T> as serde_core::de::DeserializeSeed>::deserialize::h7d89aa3704ac3a45
  [NEW] +3.81Ki  [NEW]     +16    anon.ac13b3cffb0e984c8eadac899797a3c1.67.llvm.8878368649288951961
  [DEL] -3.90Ki  [DEL]     -16    anon.fc864d48bd6fb31b92d502387c127e70.164.llvm.6438344938255192491
 -83.0% -4.04Ki -85.0% -4.04Ki    agent_data_plane::state::metrics::rules::get_datadog_agent_remappings::hbdc0274dbd81cbc3
  [DEL] -4.19Ki  [DEL]    -340    core::ptr::drop_in_place<core::option::Option<$LP$u32,tokio_util::sync::reusable_box::ReusableBoxFuture<core::result::Result<tokio::sync::semaphore::OwnedSemaphorePermit,tokio::sync::batch_semaphore::AcquireError>>$RP$>>::h3084f24fce0a93d5
  [DEL] -4.21Ki  [DEL]    -341    core::ptr::drop_in_place<http_body_util::combinators::map_err::MapErr<http_body_util::combinators::map_err::MapErr<http_body_util::combinators::map_frame::MapFrame<tonic::body::Body,tonic::codec::decode::Streaming<datadog_protos::agent_include::datadog::remoteagent::flare::v1::GetFlareFilesRequest>::new<tonic::body::Body,tonic_prost::codec::ProstDecoder<datadog_protos::agent_include::datadog::remoteagent::flare::v1::GetFlareFilesRequest>>::{{closure}}>,tonic::codec::decode::Streaming<datadog_protos::agent_include::datadog::remoteagent::flare::v1::GetFlareFilesRequest>::new<tonic::body::Body,tonic_prost::codec::ProstDecoder<datadog_protos::agent_include::datadog::remoteagent::flare::v1::GetFlareFilesRequest>>::{{closure}}>,tonic::status::Status::map_error<tonic::status::Status>>>::he50075b158634d51
  -2.9% -4.29Ki  -2.9% -4.29Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::hdf07f62ba689f516
  [DEL] -12.3Ki  [DEL] -12.2Ki    _<chrono::format::formatting::DelayedFormat<I> as core::fmt::Display>::fmt::hf5d2386408d8d014
 -31.8% -15.6Ki -31.9% -15.5Ki    agent_data_plane::main::_{{closure}}::h91f918e534067526
  [DEL] -17.3Ki  [DEL] -17.1Ki    _<figment::value::de::ConfiguredValueDe<I> as serde_core::de::Deserializer>::deserialize_struct::hcfe30cbe49586720
  [DEL] -32.6Ki  [DEL] -32.4Ki    agent_data_plane::internal::env::workload::RemoteAgentWorkloadProvider::from_configuration::_{{closure}}::hc52c5cf280184190
  +0.2% +70.6Ki  +0.2% +77.2Ki    TOTAL

pr-commenter · 2026-06-02T16:23:07Z

Regression Detector (Agent Data Plane)

Run ID: fc3713ff-aa95-4098-8625-60c43c4bee58
Baseline: dc62bbd2 · Comparison: 564e4576 · diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment (35)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment	goal	Δ mean %	links
otlp_ingest_logs_5mb_cpu (ignored)	cpu	⚪ +1.94	metrics profiles logs
otlp_ingest_metrics_5mb_cpu (erratic)	cpu	⚪ +1.15	metrics profiles logs
otlp_ingest_metrics_5mb_memory	memory	⚪ +1.11	metrics profiles logs
dsd_uds_100mb_3k_contexts_cpu (erratic)	cpu	⚪ +0.63	metrics profiles logs
otlp_ingest_traces_5mb_throughput	throughput	⚪ -0.09	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory	⚪ +0.06	metrics profiles logs
otlp_ingest_metrics_5mb_throughput	throughput	⚪ -0.02	metrics profiles logs
dsd_uds_100mb_3k_contexts_throughput	throughput	⚪ -0.00	metrics profiles logs
dsd_uds_512kb_3k_contexts_throughput	throughput	⚪ -0.00	metrics profiles logs
dsd_uds_1mb_3k_contexts_throughput	throughput	⚪ +0.00	metrics profiles logs
otlp_ingest_traces_ottl_filtering_5mb_throughput	throughput	⚪ +0.00	metrics profiles logs
otlp_ingest_logs_5mb_throughput (ignored)	throughput	⚪ +0.01	metrics profiles logs
dsd_uds_10mb_3k_contexts_throughput	throughput	⚪ +0.02	metrics profiles logs
quality_gates_rss_dsd_low	memory	⚪ -0.03	metrics profiles logs
quality_gates_rss_idle	memory	⚪ -0.05	metrics profiles logs
otlp_ingest_traces_ottl_transform_5mb_memory	memory	⚪ -0.12	metrics profiles logs
otlp_ingest_traces_ottl_transform_5mb_throughput	throughput	⚪ +0.15	metrics profiles logs
otlp_ingest_traces_5mb_memory	memory	⚪ -0.15	metrics profiles logs
dsd_uds_500mb_3k_contexts_memory	memory	⚪ -0.25	metrics profiles logs
dsd_uds_10mb_3k_contexts_memory	memory	⚪ -0.26	metrics profiles logs
otlp_ingest_traces_ottl_filtering_5mb_memory	memory	⚪ -0.29	metrics profiles logs
dsd_uds_500mb_3k_contexts_throughput	throughput	⚪ +0.29	metrics profiles logs
dsd_uds_512kb_3k_contexts_memory	memory	⚪ -0.33	metrics profiles logs
quality_gates_rss_dsd_medium	memory	⚪ -0.47	metrics profiles logs
dsd_uds_1mb_3k_contexts_memory	memory	⚪ -0.51	metrics profiles logs
dsd_uds_100mb_3k_contexts_memory	memory	⚪ -0.66	metrics profiles logs
dsd_uds_500mb_3k_contexts_cpu (erratic)	cpu	⚪ -0.71	metrics profiles logs
quality_gates_rss_dsd_heavy	memory	⚪ -0.85	metrics profiles logs
otlp_ingest_traces_ottl_filtering_5mb_cpu (erratic)	cpu	⚪ -1.19	metrics profiles logs
otlp_ingest_traces_ottl_transform_5mb_cpu (erratic)	cpu	⚪ -1.68	metrics profiles logs
dsd_uds_10mb_3k_contexts_cpu (erratic)	cpu	⚪ -1.89	metrics profiles logs
dsd_uds_512kb_3k_contexts_cpu (erratic)	cpu	⚪ -2.32	metrics profiles logs
otlp_ingest_traces_5mb_cpu (erratic)	cpu	⚪ -2.98	metrics profiles logs
otlp_ingest_logs_5mb_memory (ignored)	memory	⚪ -8.02	metrics profiles logs
dsd_uds_1mb_3k_contexts_cpu (erratic)	cpu	🟢 -8.54	metrics profiles logs

Bounds Checks: ✅ Passed (5)

experiment	check	replicates	observed	links
quality_gates_rss_dsd_heavy	memory_usage	10/10	✅ 126 MiB ≤ 140 MiB	metrics profiles logs
quality_gates_rss_dsd_low	memory_usage	10/10	✅ 40.2 MiB ≤ 50 MiB	metrics profiles logs
quality_gates_rss_dsd_medium	memory_usage	10/10	✅ 60.1 MiB ≤ 75 MiB	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory_usage	10/10	✅ 184 MiB ≤ 200 MiB	metrics profiles logs
quality_gates_rss_idle	memory_usage	10/10	✅ 26.7 MiB ≤ 40 MiB	metrics profiles logs

Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression.

…lification bug Sample forwarder_storage_max_size_in_bytes 50/50 on/off with forwarder_storage_path on a persistent compose volume, so the on-disk retry queue and restart-recovery paths run for the first time. BUG this branch surfaces: with persistence on, a network partition fills the disk-backed retry queue, and the forwarder logs error! per failed retry attempt (io.rs:462/472/421). Over a large backlog that is unbounded log amplification — it floods per-moment output, tripping 'very high output ... fail to materialize' at cx=134896 on run 4ecf6d1b, which masks other findings. The same path also opens the non-atomic torn-write hunt at persisted.rs:184 under node termination.

dd-octo-sts Bot added the area/test All things testing: unit/integration, correctness, SMP regression, etc. label Jun 2, 2026

This was referenced Jun 2, 2026

enhancement(antithesis): Vary datadog.yaml in test/antithesis, assert aliveness #1779

Merged

enhancement(antithesis): DogStatsD load generation #1782

Open

enhancement(antithesis): Expand SDK assertions, hammer sketch #1785

Open

blt mentioned this pull request Jun 2, 2026

test(antithesis): floor buffer size, multi-value packed metrics #1799

Open

4 tasks

blt force-pushed the blt/antithesis-workload-sampling branch from dd0c580 to 6945527 Compare June 2, 2026 16:26

blt force-pushed the blt/antithesis-disk-persistence branch from 04cada1 to e3b7d43 Compare June 2, 2026 16:27

blt force-pushed the blt/antithesis-workload-sampling branch from 6945527 to 6e47ff1 Compare June 2, 2026 20:16

blt force-pushed the blt/antithesis-disk-persistence branch from e3b7d43 to 1d74c55 Compare June 2, 2026 20:16

blt force-pushed the blt/antithesis-workload-sampling branch from 6e47ff1 to b291254 Compare June 2, 2026 20:33

blt force-pushed the blt/antithesis-disk-persistence branch 2 times, most recently from 1a275ee to 52a95f4 Compare June 2, 2026 20:48

blt force-pushed the blt/antithesis-workload-sampling branch 2 times, most recently from 2ac1e74 to 90323e5 Compare June 2, 2026 21:00

blt force-pushed the blt/antithesis-disk-persistence branch 2 times, most recently from 1925259 to 709025d Compare June 2, 2026 21:27

blt force-pushed the blt/antithesis-workload-sampling branch 2 times, most recently from 5a9453c to e5880cc Compare June 2, 2026 21:37

blt force-pushed the blt/antithesis-disk-persistence branch from 709025d to 00d479d Compare June 2, 2026 21:37

blt force-pushed the blt/antithesis-workload-sampling branch from e5880cc to 3662eca Compare June 2, 2026 22:26

blt force-pushed the blt/antithesis-disk-persistence branch from 00d479d to 9c8062a Compare June 2, 2026 22:26

blt force-pushed the blt/antithesis-workload-sampling branch from 3662eca to 648ccda Compare June 2, 2026 22:47

blt force-pushed the blt/antithesis-disk-persistence branch from 9c8062a to 6c92cb7 Compare June 2, 2026 22:47

blt force-pushed the blt/antithesis-workload-sampling branch from 648ccda to ed10986 Compare June 2, 2026 23:36

blt force-pushed the blt/antithesis-disk-persistence branch from 6c92cb7 to 564e457 Compare June 2, 2026 23:36

blt changed the title ~~test(antithesis): enable forwarder disk persistence — flags a log-amplification bug~~ test(antithesis): enable forwarder disk persistence Jun 3, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test(antithesis): enable forwarder disk persistence#1800

test(antithesis): enable forwarder disk persistence#1800
blt wants to merge 1 commit into
blt/antithesis-workload-samplingfrom
blt/antithesis-disk-persistence

blt commented Jun 2, 2026 •

edited

Loading

Uh oh!

blt commented Jun 2, 2026 •

edited

Loading

Uh oh!

datadog-prod-us1-4 Bot commented Jun 2, 2026 •

edited by datadog-official Bot

Loading

Uh oh!

pr-commenter Bot commented Jun 2, 2026 •

edited

Loading

Uh oh!

pr-commenter Bot commented Jun 2, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

blt commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Change Type

How did you test this PR?

References

Uh oh!

blt commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-prod-us1-4 Bot commented Jun 2, 2026 • edited by datadog-official Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

Uh oh!

pr-commenter Bot commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Binary Size Analysis (Agent Data Plane)

✅ Binary size difference within threshold

Uh oh!

pr-commenter Bot commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Regression Detector (Agent Data Plane)

Optimization Goals: ✅ No significant changes detected

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

blt commented Jun 2, 2026 •

edited

Loading

blt commented Jun 2, 2026 •

edited

Loading

datadog-prod-us1-4 Bot commented Jun 2, 2026 •

edited by datadog-official Bot

Loading

pr-commenter Bot commented Jun 2, 2026 •

edited

Loading

pr-commenter Bot commented Jun 2, 2026 •

edited

Loading